Outputting map-reduce jobs to an archive file

ABSTRACT

Method and system are provided for writing output from map-reduce jobs to an archive file. The method may include providing an archive manager and exposing an interface to be called from map-reduce jobs to output to an archive file in a map-reduce distributed file system. The method may also include using a buffering database as a temporary cache to buffer updates to the archive file. Handling by the archive manager calls from map-reduce jobs may allow: reading directly from an archive file or from a job index at the buffering database; writing to a job index at the buffering database used as a temporary cache to buffer updates; and serializing updates from the buffering database to the archive file.

BACKGROUND

This invention relates to the field of map-reduce jobs. In particular,the invention relates to outputting map-reduce jobs to an archive file.

Map-reduce frameworks such as Apache's Hadoop (Apache and Hadoop aretrademarks of The Apache Software Foundation) are well suited to readingand writing large quantities of data, using a cluster of machines to runmap-reduce jobs, process the data, and provide a distributed file systemto store data files. Map-reduce frameworks are designed to be able toscale to process more data without slowing performance. This is achievedby adding machines on which to run in parallel more instances of map orreduce tasks which can process the data in parallel.

Although distributed file systems allow map-reduce tasks to efficientlyperform concurrent reads on a single file opened on the distributed filesystem, it is not possible for multiple map-reduce tasks running withina map-reduce job to concurrently update a single file stored on thedistributed file system. For example, it is not possible for amap-reduce task to lock a region of a distributed file system file toupdate it.

A consequence of this is that it is difficult for a map-reduce job to beable to scale well and store results into a single output archive file(for example, a zip formatted file) that is portable and can easily beread by a wide variety of applications.

In data mining, there is just such a use case for building a “split”model on a big dataset, where the split model consists of an archivethat consists of 100s of thousands or even millions of individual modelfiles.

Typical approaches used in known map-reduce frameworks for scalableoutput are to: (i) store results in a distributed database system suchas a NoSQL (Not Only Structured Query Language) database, which allowsconcurrent update; or (ii) spread the output across multiple distributedfile system files (where each map-reduce task writes a separate file).However, neither of these techniques outputs a single archive file whichis easy for other applications to consume.

SUMMARY

According to a first aspect of the present invention a method foroutputting map-reduce jobs to an archive file is provided. The methodmay include providing an archive manager and exposing an interface to becalled from map-reduce jobs to output to an archive file in a map-reducedistributed file system The method may also include using a bufferingdatabase as a temporary cache to buffer updates to the archive file. Themethod may further include handling by the archive manager calls frommap-reduce jobs to allow: reading directly from an archive file or froma job index in the buffering database. The method may additionallyinclude writing to a job index in the buffering database used as atemporary cache to buffer updates. The method may include outputtingupdates from a job index to an archive file.

In further features, handling by the archive manager calls frommap-reduce jobs may include: receiving a read call for a task of amap-reduce job; connecting to the buffering database; looking up aunique token for a map-reduce job at a pending index and a committedindex provided at the buffering database; depending on the status of thejob, either reading from the archive file or reading from a job indexprovided at the buffering database.

In further features, handling by the archive manager calls frommap-reduce jobs may include: receiving a write call for a task of amap-reduce job; connecting to the buffering database; looking up aunique token for a map-reduce job at a pending index provided at thebuffering database; writing to a job index provided at the bufferingdatabase.

The archive manager may manage access to the archive file, including:allowing only one map-reduce job to open an archive file for writingupdates at a time and committing the updates on completion of a job;allowing reading from the archive file by all jobs but without allowingreading of uncommitted writes.

The method may include: maintaining a pending index at the bufferingdatabase to be accessed by a map-reduce job, whereby the pending indexincludes keys of archive file paths and values of unique tokens, wherebya unique token is allocated to a map-reduce job that has opened thearchive file for writing; the pending index including entries forarchive files containing uncommitted updates buffered in the bufferingdatabase.

The method may further include: maintaining a committed index at thebuffering database to be accessed by a map-reduce job, whereby thecommitted index includes keys of archive file paths and values of uniquetokens, whereby a unique token is allocated to a map-reduce job that hasopened the archive file for writing; the committed index includingentries for archive files for which updates have been committed but notyet serialized to the archive file.

The method may include serializing any committed updates buffered in thebuffering database to the archive file, including mapping an archivefile path name and a job index containing updates to an archive file.

Handling calls from map-reduce jobs by the archive manager may include amap-reduce job for opening an archive file for writing including:connecting to the buffering database; creating a new unique token forthe job and associating it with a path to an archive file; creating ajob index at the buffering database for the archive file to bufferupdates to the archive file.

Handling calls from map-reduce jobs by the archive manager may include amap-reduce job for committing changes to an archive file including:connecting to the buffering database; creating a serializing job toserialize updates buffered in the job index at the buffering database,to the archive file; moving the entry for the archive path and uniquejob token to the committed index at the buffering database.

Handling calls from map-reduce jobs by the archive manager may include amap-reduce job for rollback of changes to an archive file including:connecting to the buffering database; removing an entry for the archivepath and unique job token from the pending index at the bufferingdatabase.

According to a second aspect of the present invention there is provideda system for outputting map-reduce jobs to an archive file, including:an archive manager including an interface to be called from map-reducejobs to output to an archive file in a map-reduce distributed filesystem; a buffering database providing a temporary cache to bufferupdates to the archive file; whereby the archive manager handles callsfrom map-reduce jobs to: read directly from an archive file or from ajob index at the buffering database; write to a job index at thebuffering database used as a temporary cache to buffer updates; and aserializing component for outputting updates from a job index to anarchive file.

The buffering database may include a job index to buffer updates to anarchive file for a job, and whereby the name of the job index is aunique token of the updating job.

The system may include: maintaining a pending index at the bufferingdatabase to be accessed by a map-reduce job, whereby the pending indexincludes keys of archive file paths and values of unique tokens, wherebya unique token is allocated to a map-reduce job that has opened thearchive file for writing; the pending index including entries forarchive files containing uncommitted updates buffered in the bufferingdatabase.

The system may include: maintaining a committed index at the bufferingdatabase to be accessed by a map-reduce job, whereby the committed indexincludes keys of archive file paths and values of unique tokens, wherebya unique token is allocated to a map-reduce job that has opened thearchive file for writing; the committed index including entries forarchive files for which updates have been committed but not yetserialized to the archive file.

The system may include a serializing component for serializing anycommitted updates buffered in the buffering database to the archivefile, including mapping an archive file path name and a job indexcontaining updates to an archive file.

According to a third aspect of the present invention there is provided acomputer program stored on a computer readable medium and loadable intothe internal memory of a digital computer, comprising software codeportions, when said program is run on a computer, for performing themethod steps of the first aspect of the present invention.

According to a fourth aspect of the present invention there is provideda method substantially as described with reference to the figures.

According to a fifth aspect of the present invention there is provided asystem substantially as described with reference to the figures.

The described aspects of the invention provide the advantage of enablingmap-reduce jobs to output to or update a single archive file withoutsacrificing scalability.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The subject matter regarded as the invention is particularly pointed outand distinctly claimed in the concluding portion of the specification.The invention, both as to organization and method of operation, togetherwith objects, features, and advantages thereof, may best be understoodby reference to the following detailed description when read with theaccompanying drawings.

Preferred embodiments of the present invention will now be described, byway of example only, with reference to the following drawings in which:

FIG. 1 is a schematic diagram illustrating map-reduce framework in whichthe present invention may be implemented;

FIG. 2 is a block diagram of an example embodiment of a system inaccordance with the present invention;

FIG. 3 is a flow diagram of an example embodiment of a method inaccordance with the present invention;

FIGS. 4A to 4C are flow diagrams of aspects of an example embodiment ofa method in accordance with the present invention;

FIGS. 5A and 5B are flow diagrams of further aspects of an exampleembodiment of a method in accordance with the present invention;

FIG. 6 is a flow diagram of a further aspect of an example embodiment ofa method in accordance with the present invention; and

FIG. 7 is a block diagram of an embodiment of a computer system in whichthe present invention may be implemented.

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity. Further, where consideredappropriate, reference numbers may be repeated among the figures toindicate corresponding or analogous features.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth in order to provide a thorough understanding of the invention.However, it will be understood by those skilled in the art that thepresent invention may be practiced without these specific details. Inother instances, well-known methods, procedures, and components have notbeen described in detail so as not to obscure the present invention.

Method and system are provided for enabling map-reduce jobs to output toor update a single archive file without sacrificing scalability.Typically the archive file may be a zip formatted file, but otherformats may be used.

Referring to FIG. 1, an example map-reduce system 100 is shown. Amap-reduce system 100 coordinates the processing of distributed servers,running various tasks in parallel and manages communications and datatransfers between the components of the system. Map-reduce systemsprocesses and generates large data sets with a parallel, distributedalgorithm on a cluster. The “map” procedure performs filtering andsorting, and the “reduce” procedure performs a summary operation.

FIG. 1 shows a map-reduce distributed file system 110 in which aplurality of input files 120 and a plurality of output files 130 areshown. The map-reduce system 100 splits input files 120 into splits 140which are each mapped to map tasks 150. The output of the map tasks 150are forwarded to reduce tasks 160 from which they are output to theoutput files 130. A job tracker 170 may track the jobs and theirindividual tasks.

Referring to FIG. 2, an example embodiment of the described system isshown. A map-reduce system 200 is provided which includes an archivemanager 210 for outputting map-reduce jobs to an archive file 221providing in the map-reduce system's distributed file system 220.

The system includes a buffering database 230 providing a temporary cacheto buffer updates that have not yet been written to the archive file221. The buffering database 230 includes job indexes 233 for indexingbuffered updates to the archive file 221 for jobs. The bufferingdatabase 230 includes a pending index 231 and a committed index 232 forindexing jobs.

The archive manager 210 may be provided as a library, for example,implemented as a packaged file format such as a Java .jar file (Java isa trade mark of Sun Microsystems). The archive manager 210 may expose anapplication programming interface (API) which can be called frommap-reduce jobs to read and write an archive file 221.

The archive manager 210 handles calls from map-reduce tasks, by acombination of reading directly from the archive file 221 and readingand writing to the buffering database 230. The buffering database 230may be a Not Only Structured Query Language (NoSQL) database (forexample, the HBase database).

The archive manager 210 includes a job manager 240 for handling joboperations to an archive file 221 via the buffering database 230. Thejob manager 240 includes components to handle a call to the “job open”operation 241 for opening the archive file 221 for writing, “job commit”operation 242 for committing changes to the archive file 221, and “jobrollback” operation 243 for rollback changes to the archive file 221.The archive manager 210 also includes a task manager 250 for handingread task operations 251 and write task operations 252 which can becalled by map-reduce tasks running within a job to read and write toarchive file 221 while the job is running.

The archive manager 210 includes a serializing component 260 for mappingbetween the buffering database 230 and the archive file 221.

Access to the archive file 221 is managed by the archive manager 210 sothat only one job can open the archive file 221 for writing at a time.Other jobs can read from an archive file 221 but do not see uncommittedupdates.

A job that has opened the archive file 221 for write access canconcurrently write updates to the archive file 221 from multiplemap-reduce tasks that are part of the job.

Once a job ends, the updates are committed and are immediately visibleto other jobs which use the archive manager 210 to read from the archivefile 221. Serialization of updates to the archive file 221 may takelonger but the archive file 221 will eventually be consistent.

The buffering database 230 which buffers updates must be running for anarchive file 221 to be updated via the buffering archive manager 210,but is not required when jobs simply read from the archive file 221 viathe buffering archive manager 210.

Referring to FIG. 3, a flow diagram shows a flow diagram 300 of anexample embodiment of the described method.

An archive manager is provided 301 and exposes 302 an interface to becalled from map-reduce jobs to output to an archive file in a map-reducedistributed file system.

A buffering database is used 303 as a temporary cache to buffer updatesto the archive file.

A job index is provided 304 to buffer updates to an archive file for ajob, and whereby the name of the index is a unique token assigned to theupdating job.

A pending index is provided 305 to reference pending jobs at thebuffering database. A committed index is provided 306 to referencecommitted jobs at the buffering database.

The archive manager handles 307 task reads from map-reduce jobs to readdirectly from an archive file or from a job index at the bufferingdatabase, and write to a job index at the buffering database used as atemporary cache to buffer updates.

The method includes outputting 308 updates from a job index to anarchive file by serializing updates from the buffering database to thearchive file.

An example implementation of the archive manager and the operations inits API is now described. The archive manager is referred to as themap-reduce buffering archive manager (MRBAM) library and defines thefollowing operations in its API.

The first group of operations are performed by the map-reduce job at thestart of the job (MRBAM_open) and when the job ends (MRBAM_commit andMRBAM_rollback).

-   -   MRBAM_open—open the archive for writing.    -   MRBAM_commit—commit changes to the archive file.    -   MRBAM_rollback—rollback changes to the archive file.

The remaining calls can be issued by map-reduce tasks running within thejob to read and write the archive while the job is running.

-   -   MRBAM_read—read a key from an archive and return its value.    -   MRBAM_write—write a key and value pair to the archive.

MRBAM maintains two indexes called MRBAM_PENDING and MRBAM_COMMITTED inthe buffering database. These may be NoSQL indexes if the bufferingdatabase is a NoSQL database. Keys in these indexes are the distributedfile system paths in the map-reduce system of the archive files. Valuesin these indexes contain a unique token that has been allocated by MRBAMto the job that has opened the archive file for writing. Every key-valuepair in these indexes describes an MRBAM archive that is currently beingupdated.

(Key=path to archive file, Value=unique token for job that has openedarchive file for writing).

MRBAM_PENDING index (also referred to as simply the pending index)records all archive files containing uncommitted updates buffered in thebuffering database.

MRBAM_COMMITTED index (also referred to as simply the committed index)records all archive files for which updates have been committed but notyet serialized to the archive file.

The MRBAM library implements each of its API calls as described withreference to the following flow diagrams. FIGS. 4A to 4C show themethods of the job operations of job open, job commit, job rollback.FIGS. 5A and 5B show the methods of the task operations of read andwrite.

Referring to FIG. 4A, the job open operation 410MRBAM_open(archive_path) returns either a token or null.

The operation connects 411 to the buffering database for an open jobwith an archive path. It is determined 412 if the MRBAM_PENDING index orMRBAM_COMMITTED index at the buffering database already contains anentry for the archive_path parameter.

If it the indexes do already contain an entry for the archive_pathparameter, a null token is returned 413 as the archive is already openedfor writing by another job and cannot be opened.

If the indexes do not already contain an entry for the archive_pathparameter, a new unique token is created 414 and an entry added 415 toMRBAM_PENDING index associating the archive_path with the token. Thistoken must be passed to any subsequent call to the MRBAM_write operationto update the archive. It can also be optionally passed to a call to theMRBAM_read operation (if passed, it allows a read operation to returnuncommitted updates written with this token).

A new job index is created in the buffering database 416 to bufferupdates to this archive, using the token as the name of this index.

The new token is returned 417 to the caller.

Referring to FIG. 4B, the job commit operation 420.

MRBAM_commit(archive_path, token) returns either a job-id or null.

The commit job operation connects 421 to the buffering database with thearchive path and the token for the job. It is determined 422 if thekey-value pair for the job of (archive_path, token) exists in theMRBAM_PENDING index.

If it does not exist a null value is returned 423 as the archive was notopened for writing with the given access token.

If it does exist, a MRBAM_SERIALIZER job is opened 424 to serializeupdates buffered in the buffering database to the archive file stored onthe distributed file system. The archive_path and token are passed 425as parameters to the job.

The (archive_path, token) pair are moved 426 from the MRBAM_PENDINGindex to the MRBAM_COMMITTED index.

The MRBAM_SERIALIZER job is scheduled 427 to run and return its job-idto the caller.

Referring to FIG. 4C, the job rollback operation 430MRBAM_rollback(archive_path, token) returns Boolean.

The rollback operation connects 431 to the buffering database with thearchive path and job token. It is determined 432 if the key-value pairof the job (archive_path, token) exists in the MRBAM_PENDING index.

If it does not exist, then the operation returns 433 a “false” value.

If the key-value pair does exist, the job index named after the suppliedtoken is deleted 434.

The entry is removed 435 from the MRBAM_PENDING index with archive_pathas the key value. A “true” value is returned 436.

Referring to FIG. 5A, the task read operation 510MRBAM_read(archive_path, key, token) returns a value or null.

The value for the token may be null if the job does not have the archiveopened for writing OR the job does have the archive opened for writingbut does not want to read its own uncommitted writes.

It is determined 511 if the token is not null AND the connection to thebuffering database can be made AND an entry exists in MRBAM_PENDING forarchive_path with token as the value. If these criteria are met, the jobindex named after the token is opened 512 and the value for the keyreturned if the key is found in the job index.

It is determined 513 if a connection to the buffering database can bemade AND an entry exists in MRBAM_COMMITTED for the archive_path. Ifthese criteria are met, the token is retrieved from this entry, the jobindex named after the token is opened 514 and the value for the keyreturned if found in the job index.

It is determined 515 if the archive_path exists on the distributed filesystem. If it does exist AND the file contains the specified key, read516 the value for the specified key from the file and return it to thecaller.

If any of the above criteria are not met, return 517 a null value to thecaller.

Referring to FIG. 5B, the task read operation 520MRBAM_write(archive_path, key, value, token) returns Boolean.

A connection is made to the buffering database. It is determined 521 ifthe key-value pair of (archive_path, token) exists in the MRBAM_PENDINGindex. If it does not exist, return 522 a “false” value.

If it does exist, the key-value pair is written 523 into the bufferingdatabase job index named after the token and a “true” value is returned524.

To remove an existing value from the archive file for a specified key,pass null as the value parameter to MRBAM_write.

Referring to FIG. 6, a flow diagram 600 shows a MRBAM_SERIALIZER joboperation.

MRBAM_SERIALIZER is a job that runs a single mapper task, parameterizedby the archive_path and token corresponding to the distributed filesystem path name of an archive name and name of the job index containingthe updates to be made to the archive file. This task performs thefollowing steps to write any committed updates buffered in the job indexto the archive file:

A new temporary archive file is created 601 on the distributed filesystem as a copy of the file identified by archive_path if it exists.Otherwise create a new temporary empty archive file.

Iterate 602 over all key-value pairs in the job index named after thetoken. If the value is non-null, update the temporary archive file witheach key-value pair. If the value is null, remove the key from thetemporary archive file.

Rename 603 the temporary archive file to path archive_path.

Remove 604 the entry (archive_path,token) from the MRBAM_COMMITTEDindex.

Remove 605 the buffering database index named after the token.

Map-reduce jobs which update an archive file should be designed to avoidthe following scenarios. The archive manager does not define behavior inthese scenarios:

-   -   (1) writing the same key to the same archive from different        map/reduce tasks;    -   (2) reading uncommitted updates to a key from one map/reduce        task whilst writing the key from another.

The described method and system enables a portable archive file to beoutput for storing results of a map-reduce job enabling the archive fileto be read by other applications. Irrespective of the data beingprocessed, the outcome achieved by the described method and system isportability of results from map-reduce jobs using distributed filesystems.

The described method and system control an API which is called frommap-reduce jobs to read and write to an archive file.

Referring to FIG. 7, an exemplary system for implementing aspects of theinvention includes a data processing system 700 suitable for storingand/or executing program code including at least one processor 701coupled directly or indirectly to memory elements through a bus system703. The memory elements may include local memory employed during actualexecution of the program code, bulk storage, and cache memories whichprovide temporary storage of at least some program code in order toreduce the number of times code must be retrieved from bulk storageduring execution.

The memory elements may include system memory 702 in the form of readonly memory (ROM) 704 and random access memory (RAM) 705. A basicinput/output system (BIOS) 706 may be stored in ROM 704. Software 707may be stored in RAM 705 including system software 708 includingoperating system software 709. Software applications 710 may also bestored in RAM 705.

The system 700 may also include a primary storage means 711 such as amagnetic hard disk drive and secondary storage means 712 such as amagnetic disc drive and an optical disc drive. The drives and theirassociated computer-readable media provide non-volatile storage ofcomputer-executable instructions, data structures, program modules andother data for the system 700. Software applications may be stored onthe primary and secondary storage means 711, 712 as well as the systemmemory 702.

The computing system 700 may operate in a networked environment usinglogical connections to one or more remote computers via a networkadapter 716.

Input/output devices 713 may be coupled to the system either directly orthrough intervening I/O controllers. A user may enter commands andinformation into the system 700 through input devices such as akeyboard, pointing device, or other input devices (for example,microphone, joy stick, game pad, satellite dish, scanner, or the like).Output devices may include speakers, printers, etc. A display device 714is also connected to system bus 703 via an interface, such as videoadapter 715.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Improvements and modifications can be made to the foregoing withoutdeparting from the scope of the present invention.

What is claimed is:
 1. A processor-implemented method for outputtingmap-reduce jobs to an archive file, comprising: providing, by aprocessor, an archive manager and exposing an interface to be calledfrom map-reduce jobs to output to the archive file in a map-reducedistributed file system; using a buffering database as a temporary cacheto buffer updates to the archive file; handling by the archive managercalls from map-reduce jobs to allow: reading directly from the archivefile or from a job index in the buffering database; and writing to thejob index in the buffering database used as a temporary cache to bufferthe updates; outputting the updates from the job index to the archivefile, wherein the updates are concurrently written to the archive filefrom a plurality of map-reduced tasks running within a singlemap-reduced job while the single map-reduced job is running; and whereinhandling by the archive manager calls from map-reduce jobs furthercomprises: receiving a write call for a task of a map-reduce job;connecting to the buffering database; looking up a unique token for amap-reduce job at a pending index provided at the buffering database;and writing to the job index provided at the buffering database.
 2. Themethod as claimed in claim 1, wherein handling by the archive managercalls from map-reduce jobs further comprises: receiving a read call fora task of a map-reduce job; connecting to the buffering database;looking up a unique token for the map-reduce job at a pending index anda committed index provided at the buffering database; and depending onthe status of the job, either reading from the archive file or readingfrom the job index provided at the buffering database.
 3. The method ofclaim 1, wherein the archive file is a single, zip formatted file. 4.The method as claimed in claim 1, wherein the archive manager managesaccess to the archive file, further comprises: allowing only onemap-reduce job to open the archive file for writing the updates at atime and committing the updates on completion of a job; and allowingreading from the archive file by all jobs but without allowing readingof uncommitted writes.
 5. The method as claimed in claim 1, furthercomprising: maintaining a pending index at the buffering database to beaccessed by a map-reduce job, wherein the pending index includes keys ofarchive file paths and values of unique tokens, wherein a unique tokenis allocated to a map-reduce job that has opened the archive file forwriting; and the pending index including entries for archive filescontaining uncommitted updates buffered in the buffering database. 6.The method as claimed in claim 1, further comprising: maintaining acommitted index at the buffering database to be accessed by a map-reducejob, wherein the committed index includes keys of archive file paths andvalues of unique tokens, wherein a unique token is allocated to amap-reduce job that has opened the archive file for writing; and thecommitted index including entries for archive files for which updateshave been committed but not yet serialized to the archive file.
 7. Themethod as claimed in claim 1, further comprising: serializing anycommitted updates buffered in the buffering database to the archivefile, including mapping an archive file path name and the job indexcontaining updates to the archive file.
 8. The method as claimed inclaim 1, wherein handling calls from map-reduce jobs by the archivemanager includes a map-reduce job for opening the archive file forwriting further comprises: connecting to the buffering database;creating a new unique token for the job and associating it with a pathto the archive file; and creating a job index at the buffering databasefor the archive file to buffer updates to the archive file.
 9. Themethod as claimed in claim 1, wherein handling calls from map-reducejobs by the archive manager includes a map-reduce job for committingchanges to the archive file further comprises: connecting to thebuffering database; creating a serializing job to serialize updatesbuffered in the job index at the buffering database, to the archivefile; and moving an entry for the archive path and unique job token to acommitted index at the buffering database.
 10. The method as claimed inclaim 1, wherein handling calls from map-reduce jobs by the archivemanager includes a map-reduce job for rollback of changes to an archivefile further comprises: connecting to the buffering database; andremoving an entry for the archive path and unique job token from apending index at the buffering database.
 11. A computer system foroutputting map-reduce jobs to an archive file, the computer systemcomprising: one or more processors, one or more computer-readablememories, one or more computer-readable tangible storage devices, andprogram instructions stored on at least one of the one or more storagedevices for execution by at least one of the one or more processors viaat least one of the one or more memories, wherein the computer system iscapable of performing a method comprising: an archive manager includingan interface to be called from map-reduce jobs to output to the archivefile in a map-reduce distributed file system; a buffering databaseproviding a temporary cache to buffer updates to the archive file;wherein the archive manager handles calls from map-reduce jobs to: readdirectly from the archive file or from a job index at the bufferingdatabase; write to the job index at the buffering database used as atemporary cache to buffer the updates; outputting the updates from thejob index to the archive file, wherein the updates are concurrentlywritten to the archive file from a plurality of map-reduced tasksrunning within a single map-reduced job while the single map-reduced jobis running; and wherein handling by the archive manager calls frommap-reduce jobs further comprises: receiving a write call for a task ofa map-reduce job; connecting to the buffering database; looking up aunique token for a map-reduce job at a pending index provided at thebuffering database; and writing to the job index provided at thebuffering database.
 12. The system as claimed in claim 11, wherein thebuffering database includes the job index to buffer updates to thearchive file for a job, and wherein the name of the job index is aunique token of an updating job.
 13. The system as claimed in claim 11,further comprising: maintaining a pending index at the bufferingdatabase to be accessed by a map-reduce job, wherein the pending indexincludes keys of archive file paths and values of unique tokens, whereina unique token is allocated to a map-reduce job that has opened thearchive file for writing; and the pending index including entries forarchive files containing uncommitted updates buffered in the bufferingdatabase.
 14. The system as claimed in claim 11 further comprising:maintaining a committed index at the buffering database to be accessedby a map-reduce job, wherein the committed index includes keys ofarchive file paths and values of unique tokens, wherein a unique tokenis allocated to a map-reduce job that has opened the archive file forwriting; and the committed index including entries for archive files forwhich updates have been committed but not yet serialized to the archivefile.
 15. The system as claimed in claim 11, further comprising:serializing any committed updates buffered in the buffering database tothe archive file, including mapping an archive file path name and thejob index containing updates to the archive file.
 16. A computer programstored on a non-transitory computer readable medium and loadable intothe internal memory of a digital computer, comprising software codeportions, when said program is run on a computer, for performing amethod for outputting map-reduce jobs to an archive file comprising:program instructions to provide an archive manager and expose aninterface to be called from map-reduce jobs to output to the archivefile in a map-reduce distributed file system; program instructions touse a buffering database as a temporary cache to buffer updates to thearchive file; program instructions to handle by the archive managercalls from map-reduce jobs to allow: program instructions to readdirectly from the archive file or from a job index in the bufferingdatabase; and program instructions to write to the job index in thebuffering database used as a temporary cache to buffer the updates;program instructions to output the updates from the job index to thearchive file, wherein the updates are concurrently written to thearchive file from a plurality of map-reduced tasks running within asingle map-reduced job while the single map-reduced job is running; andwherein handling by the archive manager calls from map-reduce jobsfurther comprises: program instructions to receive a write call for atask of a map-reduce job; program instructions to connect to thebuffering database; program instructions to look up a unique token for amap-reduce job at a pending index provided at the buffering database;and program instructions to write to the job index provided at thebuffering database.
 17. The computer program as claimed in claim 16,wherein handling by the archive manager calls from map-reduce jobsfurther comprises: program instructions to receive a read call for atask of a map-reduce job; program instructions to connect to thebuffering database; program instructions to look up a unique token forthe map-reduce job at a pending index and a committed index provided atthe buffering database; and depending on the status of the job, programinstructions to either read from the archive file or read from the jobindex provided at the buffering database.
 18. The computer program ofclaim 16, wherein the archive file is a single, zip formatted file. 19.The computer program as claimed in claim 16, wherein the archive managermanages access to the archive file, further comprises: programinstructions to allow only one map-reduce job to open the archive filefor writing the updates at a time and committing the updates oncompletion of a job; and program instructions to allow reading from thearchive file by all jobs but without allowing reading of uncommittedwrites.
 20. The computer program as claimed in claim 16, furthercomprising: program instructions to maintain a pending index at thebuffering database to be accessed by a map-reduce job, wherein thepending index includes keys of archive file paths and values of uniquetokens, wherein a unique token is allocated to a map-reduce job that hasopened the archive file for writing; and the pending index includingentries for archive files containing uncommitted updates buffered in thebuffering database.